
United States Patent and Trademark Office 



UNITED STATES DEPARTMENT OF COMMERCE 
United States Patent and Trademark Oflice 

Address: COMMISSIONER FOR PATENTS 
P.O. Box 1450 

Alexandria, Virginia 22313-1450 
www.usplo.gov 



APPLICATION NO. 



FILING DATE 



FIRST NAMED INVENTOR 



ATTORNEY DOCKET NO. 



CONFIRMATION NO. 



09/404,940 



27662 



09/23/1999 



10/29/2003 



7590 

LYON & HARR, LLP 

300 ESPLANADE DRIVE, SUITE 800 

OXNARD, CA 93036 



ICENTARO TOYAMA 



1018.034US1 



8935 



EXAMINER 



BOOKER, KELVIN E 



ART UNIT 



PAPER NUMBER 



2121 

DATE MAILED: 10/29/2003 



't 



Please find below and/or attached an Office communication concerning this application or proceeding. 



PTO-90C (Rev. 10/03) 



Office Action Summary 



Application No. 

09/404,940 



Examiner 

Kelvin E Booker 



m 

Applicant(s) 

TOYAMA, KENTARO 



Art Unit 

2121 



-- The MAILING DATE of this communication appears on the cover sheet with the correspondence address 
Period for Reply 

A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 1 33). 

- Any reply received by the Office later than three months after the mailing date of this communication, even If timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1 .704(b). 

Status 

1 )K Responsive to communication(s) filed on iO October 2003 . 
2a)\3 This action is FINAL. 2b)S This action is non-final. 

3) n Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 11, 453 O.G. 213, 
Disposition of Claims 

4) K Claim(s) 1-31 is/are pending in the application. 

4a) Of the above claim(s) 2,3 and 30 is/are withdrawn from consideration. 

5) 0 Claim(s) is/are allowed. 

6) ^ Claim(s) 1A-29 and 31 is/are rejected. 
/)□ Claim(s) is/are objected to. 

8) n Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) 0 The specification is objected to by the Examiner, 

10)0 The drawing(s) filed on is/are: a)0 accepted or b)\Z\ objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1, 85(a). 
11 )□ The proposed drawing correction filed on is: a)n approved b)^ disapproved by the Examiner. 

If approved, corrected drawings are required in reply to this Office action. 

12) n The oath or declaration is objected to by the Examiner, 
Priority under 35 U.S.C. §§ 119 and 120 

13) D Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 119(a)-(d) or (f). 

a)nAII b)n Some*c)n None of: 

1 .□ Certified copies of the priority documents have been received. 

2. n Certified copies of the priority documents have been received in Application No. . 

3. n Copies of the certified copies of the priority documents have been received in this National Stage 

application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 

14) n Acknowledgment is made of a claim for domestic priority under 35 U.S.C. § 1 19(e) (to a provisional application). 

a) D The translation of the foreign language provisional application has been received. 

15) 0 Acknowledgment is made of a claim for domestic priority under 35 U.S.C. §§ 120 and/or 121. 
Attachment(s) 

1 ) ^ Notice of References Cited (PTO-892) 4) □ Interview Summary (PTO-413) Paper No(s). . 



2) O Notice of Draftsperson's Patent Drawing Review (PTO-948) 5) □ Notice of Informal Patent Application (PTO-152) 

3) □ Infomriation Disclosure Statement(s) (PTO-1449) Paper No(s) . 6) ^ Other: Detailed Office Action . 
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Application/Control Number: 09/404,940 
Art Unit: 2121 

DETAILED ACTION 

Response to Amendment 

1 . In the After-Final Amendment "C", filed October 10, 2003 (see paper no. 1 8), claim 30 has 
been canceled and claim 31 has been amended to depend upon claim one. Claims 1, 4-29 and 31 
are presented for further consideration. 

Allowable Subject Matter 

2. The indicated allowability of claims 1, and 4-29 is withdrawn in view of the newly 
discovered reference to Yasuda et al., "Application of Neural Network to Aesthetic Design of 
Bridges". Rejections based on the newly cited reference(s) follow. 

Claim Rejections - 35 USC § 103 

3. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all obviousness 
rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 1 02 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

4. Claims 1, 4-29 and 31 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Jagielski, "An Application of Neural Networks to Emulation of Aesthetic Judgments" [hereafter 
Jagielski] in view of Yasuda et al., "Application of Neural Network to Aesthetic Design of 
Bridges" [hereafter Yasuda]. 
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As per claim 1, Jagielski teaches of a computer-implemented method comprising: (A) 
inputting a training set including a plurality of images and a corresponding plurality of aesthetic 
scores for the images (see Abstract; and page 336, column 1: inputting images and corresponding 
aesthetic values); (B) training a classifier to provide aesthetic scores based on the training set 
(see page 336, columns 1 and 2: training the classifier); and (C) generating an aesthetic score for 
the image based on the classifier (see page 336, columns 1 and 2: training the classifier and 
generating a score). Although Jagielski clearly teaches of using back-propagation in order to 
fine-tune the disclosed neural network for several test cases (see page 336, column 2, paragraph 
four through page 337, column one, paragraph one; and Tab 2 on page 337), the art fails to 
explicitly disclose the generation of a recommendation to improve the aesthetic score. 

However, Yasuda teaches of generating a recommendation to improve an aesthetic score 
for the image by evaluating the end results and compiling a recommendation that can be applied 
to a design to improve the project (see sections 5-6 on pages 540-541, especially section 6 on 
page 541). 

It would have been obvious to one of ordinary skill in the art, at the time of the 
applicant's invention, to combine Jagielski's method of generating an aesthetic score for input 
images based on a trained neural network, with Yasuda' s method of generating a 
recommendation based upon results fi-om the network, in order to provide a method for 
presenting to the user and/or designer, a recommendation as to some of the aesthetic options, 
which when applied to an image/project, will provide greater aesthetic value. Merging the two 
methods also provides the user and/or designer the flexibility of either adding or deleting 
determined options to better refine an image/project. 
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As per claim 4, Jagielski teaches of generating aesthetic scores (see claim one above), 
but fails to explicitly disclose generating recommendations by employing gradient ascent. 

However, Yasuda teaches of generating a recommendation that comprises the use of a 
gradient ascent during the system configuration by the neural network (see section 4.1 on page 
537: quantifying the multiple inputs into the neural network in order to facilitate rule generation 
for the resultant recommendation). 

It would have been obvious to one of ordinary skill in the art, at the time of the 
applicant's invention, to combine Jagielski' s method of generating aesthetic scores with 
Yasuda's method of employing a grarf/ew^ ascent in order to provide a method for quantifying 
objective and subjective (e.g., qualitative and quantitative) input values, and generating a 
quantitative value in order to provide a numerical basis for the neural network to operate. 

As per claim 5, Jagielski teaches of generating aesthetic scores (see claim one above), 
but fails to explicitly disclose generating recommendations via local searches. 

However, Yasuda teaches of generating a recommendation comprising of performing a 
local search (see section 3 on page 535; and tables 1-3 on page 536: searching and employing 
aesthetic parameters along with local images). 

It would have been obvious to one of ordinary skill in the art, at the time of the 
applicant's invention, to combine Jagielski 's method of generating aesthetic scores with 
Yasuda's method of performing local searches in order to provide a more efficient [via both cost 
and resource usage] method of gathering information relative to a project, and forming a basis 
for generating the resulting recommendation. 
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As per claim 6, Jagielski teaches of a method wherein training a classifier comprises 
training one of a Bayesian classifier, a Support Vector Machine (S VM) classifier, a neural net 
classifier, and a decision tree classifier (see Abstract; and page 336, column 1 : employing a 
neural network for training purposes). 

As per claim 7, Jagielski teaches of a method wherein training a classifier comprises 
utilizing feature selection to correlate at least one image feature of the images with their 
corresponding aesthetic scores (see page 335, column 1: using aesthetic descriptors and 
preferences). 

As per claim 8, Jagielski teaches of a method wherein utilizing feature selection to 
correlate at least one image feature comprises utilizing feature selection to correlate at least one 
image feature selected from the group essentially consisting of: color presence, color 
distribution, geometrical quantities of segmented image parts, coefficients of image 
transformations, and higher-level image representations (see page 335, column 1: using aesthetic 
descriptors and preferences). 

As per claim 9, Jagielski teaches of a computer-implemented method comprising: (A) 
inputting an image (page 336, column 1); and (B) generating an aesthetic score for the image by 
utilizing a classifier previously trained on a training set including a pluraUty of images and a 
corresponding plurality of aesthetic scores for the images (see page 336, columns 1 and 2). 

Although Jagielski clearly teaches of using back-propagation in order to fine-tune the 
disclosed neural network for several test cases (see page 336, column 2, paragraph four through 
page 337, column one, paragraph one; and Tab 2 on page 337), the reference fails to exphcitly 
disclose the generation of a recommendation to improve the aesthetic score. 
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However, Yasuda teaches of generating a recommendation to improve an aesthetic score 
for the image by evaluating the end results and compiling a recommendation that can be applied 
to a design to improve the project (see sections 5-6 on pages 540-541, especially section 6 on 
page 541). 

It would have been obvious to one of ordinary skill in the art, at the time of the 
applicant's invention, to combine Jagielski's method of generating an aesthetic score for input 
images based on a trained neural network", with Yasuda' s method of generating a 
recommendation based upon results from the network, in order to provide a method for 
presenting to the user and/or designer, a recommendation as to some of the aesthetic options, 
which when applied to an image/project, will provide greater aesthetic value, while also 
presenting to the user and/or designer the flexibility to either add or delete aforementioned 
options. 

As per claim 10, Jagielski teaches of a method wherein generating an aesthetic score 
comprises generating an aesthetic score based on at least one image feature of the image (see 
page 335, column 1). 

As per claim 11, Jagielski teaches of a method wherein generating an aesthetic score 
based on at least one image feature of the image comprises generating an aesthetic score based 
on at least one image feature selected from the group essentially consisting of: color presence, 
color distribution, geometrical quantities of segmented image parts, coefficients of image 
transformations, and higher-level image representations (see page 335, column 1). 
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As per claim 12, Jagielski teaches of a method wherein utilizing a classifier comprises 
utilizing one of a Bayesian classifier, a Support Vector Machine (SVM) classifier, a neural net 
classifier, and a decision tree classifier (Abstract; and page 336, column 1). 

As per claims 13-18, the same limitations are subjected to in claims 1 and 4-8, therefore 
the same rejections apply (see claims 1, and 4-8 above). 

As per claim 19, the same limitations are subjected to in claim 1, therefore the same 
rejections apply (see claim 1 above). 

As per claims 20-21, the same limitations are subjected to in claims 6-7, respectively, 
therefore the same rejections apply (see claims 6-7 above). 

As per claims 22-23, the same limitations are subjected to in claims 9-10, respectively, 
therefore the same rejections apply (see claims 9-10 above). 

As per claim 24, the same limitations are subjected to in claim 12, therefore the same 
rejections apply (see claims 12 above). 

As per claims 25-27, the same limitations are subjected to in claims 1, 4 and 5, 
respectively, therefore the same rejections apply (see claims 1, 4 and 5 above). 

As per claims 28 and 29, the same limitations are subjected to in claims 16 and 18, 
respectively, therefore the same rejections apply (see claims 16 and 18 above). 

As per claim 30, Jagielski teaches of generating aesthetic scores (see claim one above), 
but fails to explicitly disclose generating recommendations based on manipulating visual 
elements in the image. 
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However, Yasuda teaches of generating recommendations on how to improve the 
aesthetic score by manipulating visual elements/characteristics in an image (see sections 5-6 on 
pages 540-541, especially section six on page 541). 

It would have been obvious to one of ordinary skill in the art, at the time of the 
applicant's invention, to combine Jagielski's method of generating an aesthetic score for input 
images based on a trained neural network, with Yasuda' s method of generating a 
recommendation based upon manipulating visual elements in an image, in order to provide a 
method for presenting to the user and/or designer, a recommendation as to some of the aesthetic 
options, which when applied to a project, will provide greater aesthetic value. 

Conclusion 

5. The following is prior art made of record and not relied upon is considered pertinent to 
appHcanfs disclosure: 

A. Laaksonen et al., "PicSOM: Self-Organizing Maps for Content-Based Image 
Retrieval"; 

B. Parisi et al., "Car Plate Recognition by Neural Networks and Image Processing"; 

C. Wong et al., "A Two-Level Model-Based Object Recognition Technique"; 

D. Alwis et al., "Searching Image Databases Containing Trademarks"; and 

E. Machado et al., "Generation and Evaluation of Artworks". 



Application/Control Number: 09/404,940 



Page 9 



Art Unit: 2121 

6. An inquiry concerning this communication or earlier communications from the examiner 
should be directed to Kelvin Booker whose telephone number is (703) 308-4088. The examiner 
can normally be reached on Monday-Friday from 7:00 AM-5:30 PM EST. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Anil Khatri, can be reached on (703) 305-0282. The fax number for the organization 
where this appUcation or proceeding is assigned is (703) 872-9306. 

An inquiry of a general nature or relating to the status of this application proceeding 
should be directed to the receptionist whose telephone number is (703) 305-3900. 
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